Image generation device

ABSTRACT

An imaging signal obtained during pattern light projection and an imaging signal during pattern light non-projection are separated from an imaging signal, and the projection angle of a light spot is specified on the basis of an array of light spots in the projection image component. The projected pattern light includes a plurality of cells that accompanies the light spots and that constitutes a discrimination code, and specifies the position of the light spots accompanied by the discrimination rode in the projection pattern on the basis of the discrimination code. Distance information to an object to be imaged can be acquired using a low amount of computation.

TECHNICAL FIELD

The present invention relates to an image generation device that canacquire, in association with a captured image, information aboutdistances to objects present in the imaged space.

BACKGROUND ART

As a conventional vehicle peripheral monitoring device, there is a knowndevice in which, when an illumination detection means detectsilluminance above a predetermined level, a display device displays thestate of a monitored area imaged by an imaging means, and when theilluminance is below the predetermined level, the display devicedisplays obstacle information acquired by illumination with patternlight, imaging of the reflected light, and data processing (see, forexample, patent reference 1).

A system that performs optical distance measurement by use of specklepatterns is also known. This system projects a primary speckle patternfrom an illumination assembly into a target area, captures a pluralityof reference images of the primary speckle pattern at differentdistances in the target area from the illumination assembly, captures atest image of the primary speckle pattern projected onto the surface ofan object in the target area, compares the test image with the referenceimages to identify the reference image in which the primary specklepattern most closely matches the primary speckle pattern in the testimage, and estimates the position of the object on the basis of thedistance of the identified reference image from the illuminationassembly (for example, patent reference 2).

PRIOR ART REFERENCES Patent References

-   Patent reference 1: Japanese Patent Application Publication No.    H6-87377 (page 2, claim 1)-   Patent reference 2: Japanese Patent Application Publication    (translation of PCT application) No. 2009-528514 (paragraphs 0001,    0006, and 0007)-   Patent reference 3: Japanese Patent Application Publication No.    2007-17643 (paragraphs 0003 and 0004)

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

A problem in the above conventional vehicle peripheral monitoring devicehas been that since its operation must be switched according to theilluminance, only one type of information can be obtained, either acaptured image or distance information.

A problem in the above conventional optical distance measurement systemhas been the need to perform correlation calculations for patternmatching between the imaged pattern and a plurality of referencepatterns, requiring a large amount of computation.

The present invention addresses these problems, and its object is toenable enabling both a captured image and distance information to beacquired, even in a bright illuminance environment, and to make itpossible to determine the position, in the projected pattern, of eachspot in the captured image and to generate an image with distanceinformation, with a small amount of computation.

Means for Solving the Problem

An image generation device according to a first aspect of the inventioncomprises:

a projection unit for projecting pattern light of a prescribedwavelength into an imaged space;

an imaging unit for imaging the imaged space;

a separation unit for separating a projected pattern image componentfrom an imaging signal obtained by imaging by the imaging unit by takinga difference between the imaging signal obtained when the pattern lightis projected and the imaging signal obtained when the pattern light isnot projected; and

a distance information generation unit for generating distanceinformation on a basis of a projected image component separated by theseparation unit; wherein

the distance information generation unit determines projection angles oflight spots in the imaged projected pattern from an arrangement of thelight spots in the projected image represented by the projected imagecomponent and a prestored relationship between the projection angles andpositions of the light spots in the projected pattern, and determines adistance to a surface of an imaged object onto which the light spots areprojected on a basis of the projection angles thus determined;

the pattern light projected from the projection unit includes aplurality of cells, each in an on state or an off state, forming anidentification code accompanying each light spot;

the distance information generation unit determines the positions of thelight spots accompanied by the identification codes in the projectedpattern on a basis of the identification codes;

the identification code accompanying each light spot has a first partcomprising a plurality of cells, aligned in a first direction in theprojected pattern, and disposed on one side of the light spot in asecond direction in the projected pattern, and a second part comprisinga plurality of cells, aligned in the second direction, and disposed onone side of the light spot in the first direction;

the identification codes accompanying the light spots that are adjacentin the first direction have at most one location at which the cellsconstituting the second parts of the identification codes change fromthe on state to the off state or from the off state to the on state; and

the first parts of the identification codes accompanying the light spotsthat are adjacent in the second direction are mutually identical.

An image generation device according to a second aspect of the inventioncomprises:

a projection unit for projecting pattern light of a prescribedwavelength into an imaged space;

an imaging unit for imaging the imaged space;

a separation unit for separating a projected pattern image componentfrom an imaging signal obtained by imaging by the imaging unit by takinga difference between the imaging signal obtained when the pattern lightis projected and the imaging signal obtained when the pattern light isnot projected; and

a distance information generation unit for generating distanceinformation on a basis of a projected image component separated by theseparation unit; wherein

the distance information generation unit determines projection angles oflight spots in the imaged projected pattern from an arrangement of thelight spots in the projected image represented by the projected imagecomponent and a prestored relationship between the projection angles andpositions of the light spots in the projected pattern, and determines adistance to a surface of an imaged object onto which the light spots areprojected on a basis of the projection angles thus determined;

the pattern light projected from the projection unit includes aplurality of cells, each in an on state or an off state, forming anidentification code accompanying each light spot;

the distance information generation unit determines the positions of thelight spots accompanied by the identification codes in the projectedpattern on a basis of the identification codes; and

the identification codes are determined in such a manner that there isonly one location at which the cells constituting the identificationcodes change from the on state to the off state or from the off state tothe on state between the light spots that are adjacent in a firstdirection in the projected pattern.

Effects of the Invention

According to the present invention, both a captured image and distanceinformation can be acquired, even in a bright illuminance environment,and the distance information associated with the image can be obtained.

In addition, the position, in the projected pattern, of each spot in thecaptured image can be determined, and information about distances toimaged objects can be acquired with a small amount of computation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an image generation device in a firstembodiment of the present invention.

FIG. 2 is a diagram that three-dimensionally illustrates the dispositionof an imaging unit 11 and a projection unit 22 in FIG. 1.

FIG. 3 is a diagram that illustrates the disposition of the imaging unitand the projection unit in the first embodiment of the invention.

FIG. 4 is a schematic diagram illustrating the configuration of theprojection unit 22 in FIG. 1.

FIG. 5 is a block diagram showing an example of the configuration of aseparation unit 16 in FIG. 1.

FIGS. 6( a) to 6(d) are diagrams that illustrate the operation of theseparation unit 16 in FIG. 5.

FIG. 7 is a block diagram showing an example of the configuration of animage generation unit in FIG. 1.

FIG. 8 is a diagram that shows an enlarged view of part of a projectedpattern.

FIG. 9 is a table showing an example of identification codes used in theprojected pattern.

FIG. 10 shows an example of the arrangement of the identification codesin the projected pattern.

FIG. 11 is a block diagram showing an example of the configuration of adistance information generation unit in FIG. 1.

FIG. 12 is a diagram that shows identification codes positioned above,below, to the left of, and to the right of a spot area.

FIG. 13 is a diagram that illustrates a procedure of the processingcarried out by the distance information generation unit in FIG. 1.

FIGS. 14( a) and 14(b) show exemplary images output by a displayprocessing unit in FIG. 1.

FIG. 15 is a diagram that illustrates identification codes accompanyingadjacent spot areas.

FIG. 16 is a block diagram showing an example of the configuration of adistance information generation unit used in a second embodiment of theinvention.

FIGS. 17( a) and 17(b) are diagrams that show the proportion in sizebetween light spots and spot areas on objects at different distances.

FIG. 18 is a diagram that illustrates a procedure of the processingcarried out by the distance information generation unit in FIG. 16.

FIG. 19 is a block diagram showing an image generation device in a thirdembodiment of the invention.

FIG. 20 is a diagram that shows an exemplary light transmissioncharacteristic of an optical filter 13 in FIG. 19.

FIG. 21 is a block diagram showing an example of the configuration of animage generation unit used in a fourth embodiment of the invention.

FIGS. 22( a) to 22(c) are diagrams that illustrate the disposition ofpixels summed by a pixel summation unit 74 in FIG. 21.

MODE FOR CARRYING OUT THE INVENTION First Embodiment

FIG. 1 is a block diagram showing the configuration of an imagegeneration device in a first embodiment of the invention. Theillustrated image generation device includes an image acquisition unit10, a pattern light generation unit 20, and a control unit 30.

The image acquisition unit 10 includes an imaging unit 11. The patternlight generation unit 20 includes a projection unit 22.

FIG. 2 three-dimensionally represents an imaged space (a space which isimaged) JS together with the projection unit 22 and the imaging unit 11.In FIG. 2, a rectangular parallelepiped object OJ1 and a sphericalobject OJ2 are supposed to be present in the imaged space JS.

The imaging unit 11 receives light from the objects OJ1, OJ2 in theimaged space JS as shown in FIG. 2 and performs imaging.

From information acquired by imaging, the image generation device in theinvention determines the distances to different parts of the objectsOJ1, OJ2, and obtains image information and information about thedistances to the different parts of the image.

As shown in FIG. 2, the projection unit 22 projects pattern light intothe imaged space JS, creating a projected pattern. In the example shownin FIG. 2, the projected pattern consists of light spots arranged in amatrix form, aligned in the lateral direction (the row direction) andthe vertical direction (the column direction). Unless otherwisespecifically noted, the term ‘direction’, when used herein in relationto the projected pattern, means a direction in the projected patternformed when the pattern light is projected onto a virtual planeperpendicular to the optical axis. This also applies when the followingdescription refers to an ‘arrangement’ or ‘position’ in the projectedpattern.

FIG. 3 is a top view of the imaging unit 11, the projection unit 22, anda light spot SP formed at an arbitrary point on one of the objects OJ1,OJ2 in the imaged space. The imaging unit 11 and the projection unit 22are spaced apart from each other by a distance Lpc in the horizontaldirection. That is, the imaging unit 11 and the projection unit 22 aredisposed at different positions in the horizontal direction, but at thesame position in the vertical direction (up and down direction). Theline linking the imaging unit 11 and the projection unit 22 is called abase line BL and the distance Lpc is called a base line length.

The lateral direction in the projected pattern corresponds to thedirection of the base line BL, that is, the horizontal direction, andthe vertical direction corresponds to the direction orthogonal to thehorizontal direction.

Suppose that light projected from the projection unit 22 forms a lightspot SP on one of the objects OJ1, OJ2 in the imaged space JS, and lightfrom the light spot SP is received by the imaging unit 11. In this case,if the projection angle φ from the projection unit 22 to the light spotSP, the incidence angle θ from the light spot SP to the imaging unit 11,and the base line length Lpc are known, the distance Z from the baseline BL to the light spot on the object OJ1 or OJ2 can be determined bya calculation based on the principle of triangulation.

Here, the projection angle φ is, as shown in FIG. 3, the angle between aline perpendicular to the base line BL and a line linking the projectionunit 22 and the light spot SP in the plane including the base line BLand the light spot SP.

The incidence angle θ is, as shown in FIG. 3, the angle between the lineperpendicular to the base line BL and a line linking the imaging unit 11and the light spot SP in the plane including the base line BL and thelight spot SP.

The incidence angle θ at the imaging unit 11 can be determined from theposition, in the imaging plane of the imaging unit 11, at which theimage of the light spot is formed, the direction of the axis line of animaging element, and the view angle.

The projection angle φ from the projection unit 22 depends on theconfiguration of the projection unit 22, and is accordingly known inadvance.

When multiple light spots are projected from the projection unit 22 atvarying projection angles and these light spots are imaged by theimaging unit 11, if the projection angles of the individual light spotsare known, the projection angle of each light spot on the imaging planecan be estimated from the relationship among the positions of the lightspots in the image.

In this case, if a condition

-   (a) ‘the magnitude relationship of the projection angles of the    individual light spots in the projection unit 22 (their order when    arranged from smaller to larger, for example) is the same as the    magnitude relationship of the incidence angles of the light spots in    the imaging unit 11 (their order when arranged from smaller to    larger)’

is satisfied and is known to be satisfied, then the projection angle ofeach of the light spots captured by the imaging unit 11 can bedetermined on this basis.

If the above condition (a) is not satisfied, or if it is not clear thatthe condition (a) is satisfied, it is necessary to determine theprojection angle of each light spot in the captured image by estimationand the like through pattern matching with captured images (referencepatterns) of projected patterns determined in advance for objectspositioned at given distances. Such processing requires an extremelylarge amount of computation, however.

This invention enables accurate estimation of the projection angles ofthe light spots captured by the imaging unit 11 with a small amount ofcomputation even when the condition (a) is not satisfied or when it isunclear that the condition (a) is satisfied.

With respect to light spots projected at varying angles in a planeperpendicular to the base line BL, the condition (a) is necessarilysatisfied for the magnitude relationship among the angles in thevertical direction, so that ‘permutations’ in the order need not beconsidered.

The case in which the base line BL extends horizontally and the lightspots in the pattern light are aligned horizontally and vertically asabove will be further described below.

Besides the projection unit 22, the pattern light generation unit 20includes a drive unit 21 as shown in FIG. 1. As shown in FIGS. 1, 3, and4, the projection unit 22 includes a laser light source 23, acollimating lens 24, an aperture 25, and a diffraction grating 26.

Under control by the control unit 30, the drive unit 21 causes the laserlight source 23 to emit light. The laser light emitted from the laserlight source 23 is converted to collimated light by the collimating lens24, and given a predetermined beam diameter by the aperture 25.

The diffraction grating 26 projects pattern light into the imaged spaceJS to generate a given projected pattern.

The imaging unit 11 includes a lens 12 and an imaging element 14, asshown in FIG. 1. The image acquisition unit 10 includes, besides theimaging unit 11, an A/D conversion unit 15, a separation unit 16, animage generation unit 17, a distance information generation unit 18, anda display processing unit 19.

The lens 12 focuses an image of the objects being imaged onto theimaging plane of the imaging element 14.

The imaging element 14 outputs imaging signals obtained by photoelectricconversion of the incident image. The imaging element 14 has a Bayerarrangement of R, G, and B pixels, for example, and outputs R, G, and Bsignals as the imaging signals.

The imaging unit 11 consisting of the lens 12 and the imaging element 14images the objects OJ1, OJ2 in the imaged space JS. This imagingoperation is performed at a given frame rate, so that multipleconsecutive frame images are obtained.

When pattern light is projected onto the objects OJ1, OJ2, (a signalrepresenting) an image in which an image (projected image component) ofthe light spots due to the projected pattern light is superimposed onthe normal light (background component) from the objects OJ1, OJ2 isoutput from the imaging unit 11.

The A/D conversion unit 15 converts the output of the imaging unit 11to, for example, an eight-bit (256-gradation) digital signal D15.

The separation unit 16 receives the output of the A/D conversion unit15, that is, the A/D-converted imaging signal D15, and separates it intothe projected image component and the background component. The imagegeneration unit 17 generates a background image from the backgroundcomponent output from the separation unit 16. The distance informationgeneration unit 18 generates distance information from the projectedimage component output from the separation unit 16.

The display processing unit 19 displays the distance informationgenerated by the distance information generation unit 18 in associationwith the background image generated in the image generation unit 17. The(signal representing the) image associated with the distance informationoutput from the display processing unit 19 is output to a display unit(not shown) or the like.

The control unit 30 controls the pattern light generation unit 20 andthe image acquisition unit 10.

The control unit 30 controls, for example, the imaging mode, frame rate,exposure time, and so on of the imaging element 14 in the imaging unit11 and sets the display mode, distance information display mode, and soon of the display processing unit 19. The control unit 30 also suppliesthe A/D conversion unit 15 with signals for controlling operationaltimings. In addition, the control unit 30 sets operating modes for thepattern light generation unit 20 and the image acquisition unit 10.

The control unit 30 also holds information Sdp indicating therelationship between identification codes (which will be describedlater) accompanying the individual light spots included in the projectedpattern projected from the projection unit 22, and positions, in theprojected pattern, of the light spots accompanied by the identificationcodes, information Spa indicating the correspondence relationshipbetween positions on the projected pattern and the projection angles,information Szv indicating the axial direction and the view angle of theimaging unit 11, and information indicating the base line length Lpc,and supplies these items of information to the distance informationgeneration unit 18.

The control unit 30 also performs control to synchronize the operationof the pattern light generation unit 20 and the operation of the imageacquisition unit 10.

More specifically, the control unit 30 controls the imaging unit 11 sothat imaging is repeated at a predetermined frame rate, whilecontrolling the drive unit 21 so that the laser light source 23 isswitched to a light emitting state and a light non-emitting state inalternate frames, and sends a signal Snf indicating whether the laserlight source 23 is in the light emitting state or the light non-emittingstate to the separation unit 16.

The frame rate of the imaging unit 11 is, for example, 30 fps, and (asignal representing) the image D11 for one frame is output from theimaging unit 11 during each frame period.

The timing of the imaging of each frame is controlled by the controlunit 30.

Since the laser light source 23 in the projection unit 22 is switchedbetween the light emitting state and the light non-emitting state atalternate frames, the projection unit 22 is switched between the stateof projecting pattern light into the imaged space JS and the state ofprojecting no pattern light at alternate frames, and the imaging unit 11can obtain images when the pattern light is projected and images when nopattern light is projected alternately, at every other frame.

From the images obtained when the pattern light is projected and theimages obtained when no pattern light is projected, the separation unit16 generates an image (the projected image component) due to the patternlight and an image (the background component) excluding the patternlight component. That is, it outputs the images obtained in the frameperiods without projected pattern light as the background component, andoutputs the images obtained by subtracting the images obtained in theframe periods without projected pattern light from the images obtainedin the frame periods with projected pattern light as the projected imagecomponent, the subtraction being made between the images obtained in theframe periods occurring one after another.

FIG. 5 is a block diagram showing an example of the configuration of theseparation unit 16.

In FIG. 5, the output (digital imaging signal) D15 of the A/D conversionunit 15 is supplied to an input terminal 60.

A frame delay unit 61 delays the digital imaging signal D15 supplied tothe input terminal 60 by one frame and outputs a frame delayed imagingsignal D61.

A difference calculation unit 62 determines the difference between theimaging signal D15 and the frame delayed imaging signal D61 (adifference obtained by subtracting the imaging signal of a frame inwhich the pattern light is not projected, from the imaging signal of aframe in which the pattern light is projected) and outputs a differencesignal D62.

A switch 63 is closed when the imaging signal D15 of a frame in whichthe projection unit 22 does not project pattern light is supplied to theinput terminal 60 and supplies the signal as the background componentD63 to the image generation unit 17 via an output terminal 65.

FIGS. 6( a) to 6(d) show an example of the operation of the separationunit 16. In the illustrated example, as shown in FIG. 6( a), the patternlight is not projected in a first frame PS1 and a third frame PS3, andthe pattern light is projected in a second frame PS2 and a fourth framePS4. As a result, captured images such as those shown in FIG. 6( b) areobtained in the individual frame periods.

In the first frame PS1, the switch 63 is closed and the imaging signalD15 at that time (the imaging signal D15 in the first frame, that is,the signal D15 obtained by digital conversion of the output D11 of theimaging unit in the state without projected pattern light) is suppliedas the background component D63 from the output terminal 65 to the imagegeneration unit 17 (FIG. 6( d)). Simultaneously, the imaging signal D15is input to the frame delay unit 61.

In the second frame PS2, the difference calculation unit 62 subtractsthe output D61 of the frame delay unit 61 (the imaging signal of thefirst frame PS1) from the imaging signal D15 at that time (the imagingsignal D15 in the second frame, that is, the signal D15 obtained bydigital conversion of the output D11 of the imaging unit in the statewith projected pattern light), and outputs the subtraction result (thedifference) D62 (FIG. 6( c)).

In the third frame PS3, as in the first frame PS1, the switch 63 isclosed, and the imaging signal D15 at that time is supplied as thebackground component D63 from the output terminal 65 to the imagegeneration unit 17. In addition, the imaging signal D15 is input to theframe delay unit 61.

In the fourth frame PS4, as in the second frame PS2, the differencecalculation unit 62 subtracts the output D61 of the frame delay unit 61from the imaging signal D15 at that time and outputs the subtractionresult (the difference) D62.

Similar processing is repeated subsequently and an image having only abackground component and an image having only a projected imagecomponent are output in alternate frame periods.

The output of the imaging element 14 is, for example, an imaging signalwith a Bayer arrangement of R pixel values, G pixel values, and B pixelvalues, and the output of the A/D conversion unit 15 is a correspondingdigital signal. But for convenience of description, FIGS. 6( a) to 6(d)show images in which pixel values are obtained for all pixels, byinterpolation.

The separation processing carried out by the frame delay unit 61, thedifference calculation unit 62, and the switch 64 in the separation unit16 is performed individually for each of R, G, and B, and interpolation(interpolation of the color components missing at each pixel in theBayer arrangement) is performed on the R, G, and B components obtainedas the result of separation, to generate all color components (R, G, andB components) for all pixels, and then the R, G, and B components ofeach pixel are combined to generate the luminance component of thepixel, and the luminance component is output as the projected imagecomponent.

The image generation unit 17 includes, for example, an image signalprocessing unit 72 as shown in FIG. 7, and applies a color interpolationprocess (interpolation of the color components missing at the positionof each pixel in the Bayer arrangement), a gradation correction process,a noise reduction process, a contour correction process, a white balanceadjustment process, a signal amplitude adjustment process, a colorcorrection process, and so on, and outputs the image obtained as theresult of these processes as the background image.

On the basis of the projected image component output from the separationunit 16 and the information about the projected pattern separatelysupplied from the control unit 30, the distance information generationunit 18 generates information indicating distances from the imaging unit11 to the individual parts of the projected image. For the generation ofthe distance information in the distance information generation unit 18,a pattern including identification codes in addition to light spots isused as the projected pattern. Accordingly, before the operation of thedistance information generation unit 18, is described, the projectedpattern used in this embodiment will be described.

The projected image (projected pattern) projected by the projection unit22 includes light spots arranged in a matrix fashion as shown in FIG. 2,as noted above and also includes, near each of the light spots, a dotgroup having a function of an identification code.

FIG. 8 shows an enlarged view of part of the projected pattern. Tosimplify the explanation, the following description will assume that theprojected pattern is projected onto a plane perpendicular to the opticalaxis of the projection unit 22.

Each of the smallest squares indicates a dot position or a cell, whichis the smallest unit in the projected pattern that can be controlled sothat it is either on (the illuminated state) or off (the non-illuminatedstate). For example, an array of cells measuring 480 rows vertically and650 columns horizontally is formed in the projection range. The cellsthat are in the illuminated state constitute the dots.

Each light spot MK is formed so as to occupy an area of cells in the onstate, measuring two rows vertically and two columns horizontally. Alight spot is also referred to as a position marker, or simply as amarker. The light spots and the dots may sometimes be collectivelyreferred to as projected points.

The row above, the row below, the column to the right, and the column tothe left of each two-row, two-column area are areas consisting of cellsin the off state (cells that are not illuminated), and the four-row,four-column area including these areas and the two-row, two-column areais referred to as a spot area MA.

The row of cells adjacent to the lower side of the four-row, four-columnspot area MA (the group of four mutually aligned dot positions adjacentto the lower side of the spot area MA) is the area constituting a firstpart DCa of the identification code. The column of cells adjacent to theright side of the spot area MA (the group of four mutually aligned dotpositions adjacent to the right side of the spot area MA) is the areaconstituting a second part DCb of the identification code. The fourcells in the first part DCa are indicated by respective referencecharacters c1-c4. The four cells in the second part DCb are indicated byrespective reference characters c5-c8.

Each of the cells in the first part DCa and the second part DCb canassume either the on-state (the illuminated state) or the off state (thenon-illuminated state), and the combination of on and off states ofthese cells constitutes an eight-bit identification code DC. Theidentification code DC accompanying each light spot MK is used toidentify the light spot MK.

The cell cbr adjacent to the right end of the first part DCa, andtherefore adjacent to the lower end of the second part DCb, is in theoff state.

The entire projected pattern is a repeated collection of areas MB, eachconsisting of five rows and five columns of cells, including a four-row,four-column spot area MA, to which an identification code DC and a cellcbr are added.

The light spots MK, which are used to determine the position of eachpart of the projected pattern, consist of a two-row, two-column array ofdots, so that they appear to occupy a relatively large area and asrelatively high brightness parts in the imaging unit 11.

The identification code DC accompanying each light spot MK is used todetermine which one among the many light spots included in the projectedpattern the light spot MK is.

FIG. 9 shows an example of the identification codes used in theprojected pattern. Fifty-six different ‘values’, that is, mutuallydiffering on/off combinations, from No. 0 to No. 55, are used in theillustrated example. The value (on or off) of each of the cells from c1to c8 constituting the identification code of each number (No.) isrepresented by ‘1’ or ‘0’.

FIG. 10 shows an exemplary arrangement of identification codes in aprojected pattern (an exemplary arrangement of areas, each consisting ofa five-row, five-column array of cells, including an identificationcode). Each square in FIG. 10 corresponds to an area MB consisting of afive-row, five-column array of cells. The number in each squareindicates the number (No.) of the identification code in FIG. 9.

In the example shown in FIG. 10, identical identification codes arelined up in the vertical direction and consecutive identification codesfrom No. 0 to No. 55 are lined up from left to right in the horizontaldirection. After (on the right side of) No. 55, next No. 0 is stationedagain, and a similar arrangement repeats (forming a cyclicalarrangement) thereafter.

The projected pattern is arranged so that No. 28 is positioned in thecenter.

When the identification codes in FIGS. 8 and 9 are arranged as in FIG.10, the array of the cells in the on-state and off-state (the array ofon-state cells and the array of off-state cells) is point symmetric withrespect to the center of the projected pattern (the center of the lightspot MK in an area MB including the identification code No. 28 locatedmidway in the vertical direction in the projected pattern).

In addition, between the identification codes accompanying the lightspots that are mutually adjacent in the horizontal direction, there isalways only one change in the on/off-state (one change from the on-stateto the off-state or one change from the off-state to the on-state).

When the projected pattern is generated by use of a diffraction grating,projecting a pattern that is point symmetric with respect to the centerof the projected pattern (a pattern that remains unchanged when rotatedby 180 degrees around its center) can simplify the form of thediffraction grating, as compared with projecting a point asymmetricalpattern, thereby reducing the design and manufacturing cost of thediffraction grating.

In consideration of this point, in this embodiment, the arrangement ofidentification codes is determined so that the projected pattern is apattern with a configuration that is point symmetric with respect to itscenter.

In addition, the number of cells in the on state in each identificationcode is four or less. The purposes of this are to facilitatediscrimination of the light spot by preventing the identification codefrom appearing as bright as the light spot, and to facilitate patterndetection by increasing the brightness of the reduced number of cells inthe on state when the image is captured.

When a projected pattern is generated by use of a diffraction pattern,if the intensity of the projected light from the light source isconstant, as the number of projected points decreases, the luminance ateach point increases, so that even if the ambient light intensity isstrong, the position of the projected pattern in the captured image caneasily be recognized. From this point of view, it is desirable for thenumber of on-state cells among the cells constituting eachidentification code to be small. In order to provide a projected patternwhich is point symmetric, and which has the fifty-six identificationcodes required for discrimination between patterns, it is also necessaryto use combinations including four cells in the on state asidentification codes.

The shape of the projected pattern formed when the pattern light isprojected onto a plane that is not perpendicular to the optical axis ofthe projection unit 22 is a quadrilateral other than a rectangle, andthe rows and the columns of light spots are not mutually parallel, andthe distances between the light spots are not uniform. In the projectedpattern formed when the pattern light is projected onto a curvedsurface, the rows and the columns of light spots fail to form straightlines. When the surface onto which the pattern light is projected isbumpy, stepped, or otherwise uneven, the magnitude relationship betweenthe incidence angles of the individual light spots (e.g., the order fromsmaller to larger) may not match the magnitude relationship between theprojection angles of the individual light spots (e.g., the order fromsmaller to larger); ‘permutations’ may occur.

In order to know the projection angle at which each light spot isprojected from the projection unit 22, it is necessary to identify thecolumn in which the light spot is located in the matrix. The eight-bitidentification code itself does not include enough information toidentify the column. However, even when the order of the light spots ispermuted, if the shift from the original position (order) of each lightspot is within the range of the cycle of change in the ‘value’ of theidentification code (the range corresponding to fifty-six areas MB, eachconsisting of cells in five rows and five columns, in the example ofFIG. 10), it is possible to identify the non-permuted position (theoriginal position). By identifying the original position, it is possibleto identify the column in which the light spot accompanied by theidentification code is located in the matrix.

‘Permutations’ occur because the imaging unit 11 and the projection unit22 are disposed at horizontally different positions. Since the imagingunit 11 and the projection unit 22 are disposed at vertically identicalpositions, such permutations do not occur in the vertical direction, sothat vertical position (order) in the projected pattern can bedetermined by detecting the order in the captured image. Therefore,codes for identifying the vertical order are unnecessary.

FIG. 11 shows an example of the configuration of the distanceinformation generation unit 18.

The distance information generation unit 18 shown in FIG. 11 includes abinarization unit 81, a spot area extraction unit 82, an identificationcode reading unit 83, a storage unit 84, a validation unit 85, aprojection angle estimation unit 86, an incidence angle calculation unit87, and a distance calculation unit 88.

The binarization unit 81 binarizes the projected image component outputfrom the separation unit 16 and outputs a binary projected image.

The spot area extraction unit 82 extracts, from the projected image,spot areas MA (the four-row, four-column areas in FIG. 8) centered onindividual light spots.

The spot areas MA are extracted by searching for four-row, four-columngroups of cells at fixed intervals, each group having four dots(on-state cells) in the middle two rows and the middle two columnssurrounded by off-state cells (in the top and bottom rows and the rightand left columns). The groups of four dots in the middle two rows andthe middle two columns are regularly spaced at equal intervals in theprojected pattern, so that the image obtained by imaging should satisfya similar condition. In the captured image, however, due to curvature,bumps, steps, or other types of unevenness, in the surface of the imagedobject, the intervals are not necessarily exactly equal, so that patternmatching or the like based on degree of similarity is performed toextract the spot areas MA.

The identification code reading unit 83 reads the identification codesDC from the identification code areas adjacent to the extracted spotareas MA.

Not only the first part DCa adjacent to the lower side and the secondpart DCb adjacent to the right side of each spot area MA are read, thepart adjacent to the upper side (the first part (indicated by referencecharacters DCa′) of the identification code for the light spot in theupper adjacent spot area) and the part adjacent to the left side (thesecond part (indicated by reference characters DCb′) of theidentification code for the light spot in the left adjacent spot area)are also read at this time. The values of the identification codes thatare read are stored in the storage unit 84.

When the identification code reading unit 83 reads the identificationcode adjacent to the each spot area MA, if the upper adjacentidentification code part (DCa′) or the left adjacent identification codepart (DCb′) has already been read and stored in the storage unit 84, thevalue of this identification code part may be read from the storage unit84. If the captured projected image is processed sequentially, startingfrom the upper left, then when the processing related to each spot areais performed, the processing of the upper and left adjacent spot areashas already been finished, so that their identification codes are storedin the storage unit 84 as described above and are available for use.

The validation unit 85 checks the validity of each identification coderead by the identification code reading unit 83. If the result of thischeck is that the validity is doubtful (unreliable), the identificationcode that was read is not used in the subsequent processing.

The validity determination uses, as shown in FIG. 12, the firstidentification code part DCa adjacently below, the second identificationcode part DCb adjacently to the right, the identification code part DCa′adjacently above, and the identification code part DCb′ adjacent to theleft of the each spot area MA.

In FIG. 12, as in FIG. 8, the states of the four cells (four bits)constituting the identification code part DCa are indicated by c1-c4 andthe four bits constituting the identification code part DCb areindicated by reference characters c5-c8.

In addition, the states of the four cells (four bits) constitutingidentification code part DCa′ are indicated by c1′-c4′, and the fourbits constituting identification code part DCb′ are indicated byreference characters c5′-c8′.

Since c1′-c4′ accompany a light spot MK aligned in the upward direction,they should have the same values as c1-c4, respectively.

Since c5′-c8′ accompany the next light spot MK to the left, according tothe condition that ‘there is always only one change in on/off-statebetween mutually adjacent identification codes’, they should have thesame values as c5-c8, or differ in only one bit.

A decision is accordingly made that if a condition that (b) ‘c1-c4 arethe same as c1′-c4′ and c5-c8 are the same as c5′-c8′ or differ in onlyone bit’

is satisfied, then the identification code c1-c8 that has been obtainedis valid, and if the condition (b) is not satisfied, then theidentification code c1-c8 that has been obtained is not valid (low inreliability).

The condition (b) can also be rephrased as

(b1) ‘there is no more than one difference (change) between the secondparts c5-c8 and c5′-c8′ of the identification codes of the light spotsadjacent in the horizontal direction, and the first parts c1-c4 andc1′-c4′ of the identification codes of the light spots adjacent in thevertical direction are mutually identical’, or as

(b2) ‘there is no more than one difference (change) between theidentification code part c5-c8 adjacent to the right side and theidentification code part c5′-c8′ adjacent to the left side of each lightspot, and the identification code part c1-c4 adjacent to the lower sideand the identification code part c1′-c4′ adjacent to the upper side arethe same.’

The above assumes that bits c1′-c8′ have already been determined to bevalid.

In the state in which the validity of bits c1′-c8′ has not yet beendetermined, it is also permissible to suspend the determination that anyone of c1-c8 and c1′-c8′ is invalid and make a comprehensivedetermination by utilizing the results of comparisons with otheridentification codes as well.

The projection angle estimation unit 86 receives the results of thereading of the identification codes from the identification code readingunit 83 and the validity check results D85 from the validation unit 85,and further obtains, from the control unit 30, data Sdp indicating thecontent of the table in FIG. 9 (information indicating the relationshipbetween the identification codes and the positions in the projectedpattern) and information Spa indicating the correspondence relationshipbetween the positions in the projected pattern and the projectionangles, based on all of which it estimates the projection angle φ ofeach light spot. When the above information, more specifically the dataSdp indicating the content of the table in FIG. 9 and the informationSpa indicating the correspondence relationship between the positions inthe projected pattern and the projection angles, is supplied from thecontrol unit 30, this information may be held in a memory (not shown) inthe projection angle estimation unit 86.

If the read result in the identification code reading unit 83 isdetermined to be invalid by the validation unit 85, the projection angleestimation unit 86 does not estimate the projection angle from the readresult.

If the read result in the identification code reading unit 83 isdetermined to be valid by the validation unit 85, the projection angleestimation unit 86 estimates the projection angle from the read result.

In the estimation of the projection angle, which one of identificationcodes No. 0 to No. 55 in the table in FIG. 9 the value of the readidentification code DC matches is determined (i.e., where the light spotto which it is assigned is positioned in the pattern is determined), andbased on the determination result, the position of the light spot in theprojected pattern is identified.

If the identification code is not present in the table in FIG. 9 (doesnot match any one of the identification codes assigned to the lightspots in the pattern), a read error is declared and the read code is notused for determining the position of the light spot.

When the position of the light spot in the projected pattern has beenidentified, the projection angle φ is determined on the basis of theinformation Spa (supplied from the control unit 30) indicating therelationship between the identified position and the projection angle.

From the output of the spot area extraction unit 82, the incidence anglecalculation unit 87 calculates the incidence angle θ of the light spoton the basis of the position in the captured image at which the lightspot is imaged and the axial direction and the view angle of the imagingunit. The information Szv indicating the axial direction and the viewangle is supplied from the control unit 30.

The distance calculation unit 88 calculates the distance to the surfaceof the imaged object onto which the light spot is projected on the basisof the projection angle φ estimated by the projection angle estimationunit 86, the incidence angle e calculated by the incidence anglecalculation unit 87, and the base line length Lpc supplied from thecontrol unit 30.

First, the distance Z from the axis line BL in FIG. 3 to the surface ofthe imaged object onto which the light spot is projected (the point atwhich the spot SP is formed), that is, the distance to the spot SP, canbe obtained from the relation:

Z=Lpc/(tan φ−tan θ)   (1)

Equation (1) is obtained from the following relation in FIG. 3:

Z·tan φ−Z·tan θ=Lpc   (2)

Next, the distance R from the imaging unit to the surface (spot SP) ofthe imaged object on which the light spot is formed can be obtained fromthe distance Z to the base line BL obtained by the equation (2) and theincidence angle θ as follows:

R=Z/cos θ  (3)

FIG. 13 illustrates a procedure of the processing carried out by thedistance information generation unit 18 in FIG. 1.

First the binarized projected image pattern is binarized (ST101).

Next, spot areas MA are extracted from the binarized projected imagepattern (ST102).

Next, identification codes DC, DCa′, and DCb′ are read from theidentification code areas adjacent to a spot area MA (ST103).

Next, the validity of the identification code is determined (ST104).

If the identification code is valid, the projection angle φ is estimated(ST105).

Next, the incidence angle θ is calculated (ST106).

Next, the distance is calculated by using the projection angle φ and theincidence angle θ (ST107).

Whether or not steps ST103 to ST107 have been performed for all the spotareas MA in the captured projected image is now determined (ST108), andif all have been processed, the process is terminated.

If a not-valid (No) decision is made in step ST104, the process proceedsto step ST108.

From the above process, the distance to the surface (spot SP) of theimaged object onto which each light spot is projected can be obtained.

In this case, even if the order of incidence angles to the imaging unitdiffers from the order of the projection angles in the projection unit(even if a permutation occurs), the use of the identification codesenables determination of the positions in the projected pattern, of thelight spots in the captured image. It is therefore possible to identifythe positions in the projected pattern, estimate the projection angles,and calculate the distances, accurately and with a small amount ofcomputation.

The display processing unit 19 displays the distance information inassociation with the background image.

FIGS. 14( a) and 14(b) show exemplary images output by the displayprocessing unit 19.

FIG. 14( a) shows a background image, and

FIG. 14( b) shows an image with distance information.

As the image with distance information, an image with brightnesses orcolors assigned to distances is displayed. For example, an image inwhich the background image is represented by brightness and the distanceis represented by color is displayed. Alternatively, an object presentin the imaged space is recognized and an image in which text informationexpressing the distance to the object is displayed, being superimposedon the background image is output.

It is also possible to use, for example, two display screens, thebackground image in FIG. 14( a) being displayed on one of them, theimage with distance information shown in FIG. 14( b) being displayed onthe other one; or the background image shown in FIG. 14( a) and theimage with distance information shown in FIG. 14( b) may be displayedalternately on one display screen; or the image selected by a useroperation may be displayed. In this case, the image with distanceinformation is preferably displayed in synchronization with thebackground image, with the same view angle and the same number ofpixels.

As described above, according to this embodiment, distance can bedetermined with a small amount of computation.

When a pattern is projected by use of a diffraction grating, a pointsymmetrical pattern can facilitate the design of the diffraction gratingand reduce its cost.

In addition, limiting the number of on-state cells in eachidentification code DC to four at most facilitates recognition of thelight spots, and reduces the number of dots (constituted of the on-statecells), so that the brightness of the dots can be increased, and patterndetection in camera imaging can be facilitated.

In the above embodiment, if the condition (b) is satisfied, theidentification code c1-c8 that has been obtained is determined to bevalid; if the condition (b) is not satisfied, the identification codec1-c8 that has been obtained is determined to be invalid (low inreliability). Alternatively, by using the fact that:

(c) there is always only one change in on/off-state (one change from theon-state to the off-state or one change from the off-state to theon-state) between the identification codes of light spots that aremutually adjacent in the horizontal direction,

the identification codes of adjacent light spots may be compared todetermine the validity of the results of the reading of theidentification codes.

For example, as shown in FIG. 15, the identification code consisting ofthe cells c1-c8 below and to the right of each light spot MK is comparedwith the identification code consisting of the cells c1-c8 below and tothe right of the left adjacent light spot MK′, and when a conditionthat:

(c1) ‘the identification code consisting of the cells c1-c8 below and tothe right of each light spot differs from the identification codeconsisting of the cells c1-c8 below and to the right of the leftadjacent light spot by one bit’

is satisfied, it may be determined that the obtained identification codec1-c8 is valid; when the condition (c1) is not satisfied, it may bedetermined that the obtained identification code c1-c8 is invalid (lowin reliability).

As an identification code DC, the above example uses a code includingthe first part DCa consisting of horizontally aligned cells adjacent tothe lower side of the spot area and the second part DCb consisting ofvertically aligned cells adjacent to the right side of the spot area.The code, however, may include only one of the first part and the secondpart. The first part may be adjacent to the upper side of the spot area.The second part may be adjacent to the left side of the spot area.Alternatively, the identification code may include just one of a partconsisting of horizontally aligned cells such as the first part and apart consisting of vertically aligned cells such as the second part.

The first embodiment uses a configuration in which the imaging unit 11and the projection unit 22 are disposed so that they are horizontallyaligned and the identification code enables a light spot to bediscriminated from light spots present at other positions in thehorizontal direction. But a configuration in which the imaging unit 11and the projection unit 22 are disposed so that they are verticallyaligned and the identification code enables a light spot to bediscriminated from light spots present at the other positions in thevertical direction may also be used.

In summary, it is only necessary to use an identification code thatenables a light spot to be discriminated from other light spots presentat different positions in the direction in which the imaging unit 11 andthe projection unit 22 are aligned (the first direction in the space inwhich the imaging unit 11 and the projection unit 22 are placed).

If the direction in the projected pattern corresponding to the directionin which the imaging unit 11 and the projection unit 22 are aligned isreferred to as the first direction and the direction perpendicular tothe first direction is referred to as the second direction, theconditions (b1) and (b2) described with reference to FIG. 12 can bestated in more general terms as:

(d1) ‘there is no more than one difference (change) between the secondparts (c5-c8 and c5′-c8′) of the identification codes of the light spotsadjacent in the first direction, and the first parts (c1-c4 and c1′-c4′)of the identification codes of the light spots adjacent in the seconddirection are identical to each other’; and

(d2) ‘for each light spot, there is no more than one difference (change)between the identification code part (c5-c8) adjacent on one side in thefirst direction and the identification code part (c5′-c8′) adjacent onthe other side in the first direction, and the identification code part(c1-c4) adjacent on one side in the second direction and theidentification code part (c1′-c4′) adjacent on the other side in thesecond direction are identical’;

the condition (c) described with reference to FIG. 15 can be representedas

(e) ‘there is only one difference between the identification codes oflight spots that are adjacent in the first direction’;

in any of these cases, whether the results of the reading of theidentification codes are valid or not is determined according to whetheror not these conditions are satisfied.

Furthermore, whether the results of the reading of the identificationcodes are valid or not may be determined according to whether or not thecondition (d) and the condition (e) are both satisfied.

Second Embodiment

The configuration of the second embodiment of the invention is shown inFIG. 1, as in the first embodiment. As the distance informationgeneration unit 18, however, the one illustrated in FIG. 16 is used.

The distance information generation unit 18 in FIG. 16 is substantiallyidentical to the one in FIG. 11, but differs in that a distance rangecalculation unit 89 is added.

The distance range calculation unit 89 estimates a range of distances tothe surface of an imaged object on which a light spot is projected, fromthe ratio between the size of the spot area MA in the captured image andthe size of the dot forming the light spot (the size of the part withrelatively high brightness in each cell in the image obtained byimaging).

If the distance calculated on the basis of the projection angle is notwithin the range of distances calculated by the distance rangecalculation unit 89, the distance calculation unit 88 infers that theresult of the calculation based on the projection angle is low inreliability.

The method used in the distance range calculation unit 89 to calculatethe range of distances will now be described with reference to FIGS. 17(a) and 17(b).

FIGS. 17( a) and 17(b) illustrate ratios between the size of the spotarea MA and the size of the dot DT constituting the light spot MK inrelation to the distance to the imaged object.

When the projected pattern is projected by use of the combination of thelaser light source and the diffraction grating, a dot in the pattern isformed by a point beam of laser light collimated by the collimatinglens, so that the size of the dot itself does not change, regardless ofthe distance to the imaged object. The size of the entire pattern, onthe other hand, depends on the mutual distances of the diffracted dots.Since the diffracted beams forming different dots are not parallel toeach other but are projected radially from the projection unit 22, thedistance between the dots widens as the distance to the imaged objectincreases. Therefore, the size of the projected pattern and thus theratio between the size of the spot area MA and the size of the dot DTvaries according to the distance to the imaged object, and by measuringthis ratio, the possible range of distances to the imaged object isfound.

The resolution (number of pixels) of the imaging element needs to behigh enough to detect changes in the ratio of the size of each dot tothe size of the spot area MA. Specifically, the number of pixels in thehorizontal direction and the number of pixels in the vertical directionin the imaging element needs to be sufficiently higher than,respectively, the number of dot positions (cells) in the horizontaldirection and the number of dot positions (cells) in the verticaldirection in the projected pattern, preferably by a factor of, forexample, about ten or more.

FIG. 18 illustrates a procedure of the processing in the distanceinformation generation unit 18 in the second embodiment.

The processing in FIG. 18 is the same as in FIG. 13, except that stepsST111 to ST114 are added.

In step ST111, a range of distances to the surface of an imaged objecton which dots are formed is estimated from the ratio between the size ofthe dot and the size of the spot area MA.

In step ST112, whether or not the distance obtained in step ST107 iswithin the range of distances obtained in step ST111 is determined. Ifit is within the range of distances (Yes), the distance obtained inST107 is decided to be valid (ST113), processing based on the result ofthis decision is performed, and the processing proceeds to step ST108.If it is not within the range of distances (No), the distance obtainedin step ST107 is decided to be invalid (ST114), processing based on theresult of this decision is performed, and the processing proceeds tostep ST107.

In regard to points other than the above, the second embodiment is thesame as the first embodiment.

As described above, by narrowing the range of possible distances to animaged object on the basis of the ratio between the size of the spotarea MA and the size of the dot DT forming the light spot, erroneousdetection of light spots can be reduced and the accuracy of detection ofthe distance to the imaged object can be enhanced.

The variations described in the first embodiment are also applicable tothe second embodiment.

Third Embodiment

FIG. 19 is a block diagram showing the configuration of an imagegeneration device in a third embodiment of the invention.

The illustrated image generation device is substantially the same as theone shown in FIG. 1, but differs in that the imaging unit 11 includes anoptical filter 13.

The optical filter 13 is a spectral filter, which operates with a givenwavelength characteristic in transmitting incident light. Specifically,it is has a characteristic having a lower transmittance in thewavelength band of visible light than at the wavelength of projectedpattern light.

The significance of using the optical filter 13 will now be described.

An exemplary spectral transmission characteristic of the optical filter13 is shown in FIG. 20. With the characteristic shown in FIG. 20, 100%transmission takes place in a wavelength band centered on 830 nm, whichis the wavelength of the projected pattern light, that is, the emissionwavelength of the laser light source 23; transmission is limited to aprescribed transmittance in the wavelength band of visible light; and notransmission takes place in other wavelength bands.

The prescribed transmittance in the visible light wavelength band is seton the basis of the spectral distribution and the brightness of theambient light in the imaged space JS and the intensity of the light inthe projected pattern projected by the projection unit 22, especiallythe light intensity of each dot.

When there is no optical filter 13, the ambient light component, whichis distributed with high power, being centered on the visible wavelengthband, is dominant in the imaging signal D11 output from the imagingelement 14, and the pattern light component, which is restricted to thewavelength band centered on 830 nm, accounts for only a tiny share,making it difficult to extract the pattern light component.

In this embodiment, therefore, the optical filter 13 is provided, andthe pattern light component is imaged with 100% transmittance, while theambient light component is imaged with attenuation, thereby tofacilitate separation or extraction of the projected image componentfrom the imaging signal.

If the ratio between the projected image component and the backgroundcomponent in the imaging signal D11 output from the imaging element 14is 1:64, then about four of the 256 gradation steps of the eight-bitimaging signal D15 obtained by the A/D conversion of the imaging signalD11 represent the projected image component.

In principle, separation is possible if the difference between thebackground component and the sum of the projected image component andthe background component is one gradation step or more. But, by allowingfor the influence of noise, the above-mentioned difference is set tohave a value equal to or greater than a prescribed number of gradationsteps. The prescribed number of gradation steps is set to a numberobtained by adding the number of gradation steps of the anticipatednoise component to the minimum one gradation step required when there isno noise.

The transmittance of the optical filter 13 in the wavelength bandvisible light is set so that the ratio between the projected imagecomponent and the background component is 1:64, for example.

The light intensity of each dot in the projected pattern depends on theemission power of the laser light source 23 and the number of dotsformed. The illuminance at each wavelength of ambient light depends onthe spectroscopic radiation characteristic of the ambient light source;the quantity of light can be calculated from the spectroscopic radiationcharacteristic.

An example in which the emission wavelength of the laser light source 23is 830 nm has been described in the third embodiment. However, any laserlight source having an emission wavelength at which the radiance of thespectral radiance characteristic of the ambient light source is weak maybe used.

Since the imaging element in the third embodiment is provided with theoptical filter having a spectral transmission characteristic, thepattern light component can be extracted by minimizing the influence ofthe dominant ambient light component, distance measurement is possibleeven in a high brightness environment, and the distance informationassociated with the image can be obtained. Since both the backgroundimage and the distance information can be obtained by use of a singleimaging element, the background image due to ambient light has exactlythe same view angle as the image giving the distance information, andthe distances to the imaged objects appearing in the image can beacquired accurately.

As the optical filter 13, a filter which has transmissivecharacteristics in the visible wavelengths band and at the wavelength ofthe projected pattern light, with the transmittance in the visiblewavelength band being set lower than the transmittance at the wavelengthof the projected pattern light is used, so that in the output of theimaging element 14, the ambient light component in the visiblewavelength band that interferes with the extraction of the pattern lightcomponent can be reduced, enabling highly precise extraction of thepattern light component, allowing distance measurement even in a brightenvironment, and making it possible to obtain the distance informationin association with the image.

In the first, second, and third embodiments, an imaging element with aBayer arrangement of RGB pixels is used as the imaging element 14. But amonochrome imaging element can also operate as described above andprovide the same effects.

Fourth Embodiment

The configuration of the fourth embodiment of the invention is shown inFIG. 19, as in the third embodiment. As the image generation unit 17,however, the one illustrated in FIG. 21 is used.

The image generation unit 17 shown in FIG. 21 includes an image signalprocessing unit 72 similar to the one shown in FIG. 7, and, in addition,a pixel summation unit 74.

The imaging signal not including the projected image component, outputfrom the separation unit 16 (the signal indicating the backgroundcomponent) is applied to an input terminal 71 in FIG. 21.

The pixel summation unit 74 adds, to the pixel value of each of the R,G, and B pixels (the pixel of interest) in the Bayer arrangement of Rpixels, G pixels, and B pixels output from the output terminal 65 of theseparation unit 16, the pixel values of pixels with the same color,located around that pixel of interest, thereby outputting a signalhaving an amplified pixel value.

FIGS. 22( a) to 22(c) show the pixels which are added to the pixel ofinterest. In FIGS. 22( a) to 22(c), each of the smallest squaresrepresents one pixel.

When the pixel of interest is the R pixel RR34 as shown in FIG. 22( a),eight pixels added as surrounding pixels are: the pixel RR12 in thesecond row above and the second column to the left, the pixel RR32 inthe second row above and the same column, the pixel RR52 in the secondrow above and the second column to the right, the pixel RR14 in the samerow and the second column to the left, the pixel RR54 in the same rowand the second column to the right, the pixel RR16 in the second rowbelow and the second column to the left, the pixel RR36 in the secondrow below and the same column, and the pixel RR56 in the second rowbelow and the second column to the right.

Accordingly, the summation result NRR34 is represented by the followingequation.

NRR 34 = RR 12 + RR 32 + RR 52 + RR 14 + RR 34 + RR 54 + RR 16 + RR 36 + RR 56

The case in which the pixel of interest is RR34 has been describedabove. With respect to the R pixels at other positions, surroundingpixels having the same positional relationship as the above are added.

When the pixel of interest is the G pixel GB33 as shown in FIG. 22( b),eight pixels added as surrounding pixels are: the pixel GB31 in thesecond row above and the same column, pixel GR22 in the first row aboveand the first column to the left, the pixel GR42 in the first row aboveand the first column to the right, the pixel GB13 in the same row andthe second column to the left, the pixel GB53 in the same row and thesecond column to the right, the pixel GR24 in the first row below andthe first column to the left, the pixel GR44 in the first row below andthe first column to the right, and the pixel GB35 in the second rowbelow and the same column.

Accordingly, the summation result NGB33 is represented by the followingexpression.

NGB 33 = GB 31 + GR 22 + GR 42 + GB 13 + GB 33 + GB 53 + GR 24 + GR 44 + GB 35

The case in which the pixel of interest is GB33 has been describedabove. With respect to the G pixels at other positions, surroundingpixels having the same positional relationship as the above are added.

When the pixel of interest is the B pixel BB43 as shown in FIG. 22( c),eight pixels added as surrounding pixels are: the pixel BB21 in thesecond row above and the second column to the left, the pixel BB41 inthe second row above and the same column, the pixel BB61 in the secondrow above and the second column to the right, the pixel BB23 in the samerow and the second column to the left, the pixel BB63 in the same rowand the second column to the right, the pixel BB25 in the second rowbelow and the second column to the left, the pixel BB45 in the secondrow below and the same column, and the pixel BB65 in the second rowbelow and second column to the right.

Accordingly, the summation result NBB43 is represented by the followingequation.

NBB 43 = BB 21 + BB 41 + BB 61 + BB 23 + BB 43 + BB 63 + BB 25 + BB 45 + BB 65

The case in which the pixel of interest is BB43 has been describedabove. With respect to R pixels at other positions, surrounding pixelshaving the same positional relationship as the above are added.

The above summing process mixes the surrounding pixels in the same framewith the pixel of interest. As the surrounding pixels generally have thesame value as the pixel of interest, the effect is to amplify the signalcomponent.

If the pixel values of eight surrounding pixels are added to each pixelof interest as described above, for example, (and if the surroundingpixels are assumed to have the same pixel value as the pixel ofinterest), the summation result is nine times the pixel value of thepixel of interest.

As a result of adding (mixing) the surrounding pixels, however, theresolution (static resolution) is degraded.

Instead of adding surrounding pixels in the same frame to the pixel ofinterest, pixels at the same position as the pixel of interest indifferent frames, that is, frames preceding and following the frameincluding the pixel of interest, may be added.

In this case, the preceding and following frames are not limited to thesingle immediately preceding frame and the single immediately followingframe; the preceding and following frames may be a given number ofimmediately preceding frames and a given number of immediately followingframes.

Adding pixels at the same position in different frames enables thesignal component to be amplified while avoiding loss of staticresolution, and is particularly effective for scenes with little motion.

In the case of scenes with increased motion, however, motion blur isincreased.

Both surrounding pixels in the same frame and pixels at the sameposition in different frames may be added to the pixel of interest, andin addition, pixels surrounding pixels at the same position in thedifferent frames may be added.

In this manner, the amplification factor of the signal component can befurther increased.

The image signal processing unit 72 applies a gradation correctionprocess, a noise reduction process, a contour correction process, awhite balance adjustment process, a signal amplitude adjustment process,a color correction process, and so on to the output signal of the pixelsummation unit 74, and outputs the resultant image signal, as thebackground image, from an output terminal 73.

As in the third embodiment, a filter having the spectral transmissioncharacteristic shown in FIG. 20 is used as the optical filter 13.However, in the present embodiment the transmittance in the visiblewavelength band is set on the basis of the number of pixels summed inthe pixel summation unit 74. For example, the transmittance is set tosmaller values as the number of pixels summed by the pixel summationunit 74 increases. More specifically, the transmittance of the opticalfilter 13 is set to the reciprocal of the number of pixels summed in thepixel summation unit 74. If the pixel summation unit 74 sums ninepixels, for example, the transmittance is set to 1/9 (11.1%).

It is sufficient if the pixel summation unit 74 is able to restore thebrightness of the image to compensate for the attenuation in the visiblewavelength band caused by the optical filter 13, while holding the lossof resolution to a minimum. Accordingly, any type of signal processingother than the addition of surrounding pixels may be applied.

For example, by detecting correlations between pixel values, andselecting and summing strongly correlated pixels, the loss of resolutionin the background image can be further reduced.

In the examples shown in FIGS. 22( a) to 22(c), the pixel summation unit74 adds eight surrounding pixels to the pixel of interest. It may be soconfigured that the transmittance of the optical filter 13 in thevisible wavelength band is set and the number of surrounding pixels thatare added is increased or decreased according to the ratio between theambient light component and the pattern light component. This makes itpossible not only to extract the pattern light component but also toobtain a bright background image.

Like the first, second, and third embodiments, the fourth embodimentalso permits the use of a monochrome imaging element, instead of animaging element with a Bayer array of R, G and B pixels, as the imagingelement 14, with similar operation and similar effects. Furthermore, theuse of a monochrome imaging element in the fourth embodiment enablespixels at closer positions to be summed, so that an image with less lossof resolution can be generated as the background image.

According to the fourth embodiment, the image generation unit 17 isprovided with the pixel summation unit 74 for mixing the surroundingpixels to amplify the signal component. Therefore, even when thequantity of incident light of the background image component enteringthe imaging element 14 is reduced by setting the transmittance of theoptical filter 13 in the visible wavelength band to a value lower thanthe transmittance of the wavelength of the pattern light, the mixing ofthe surrounding pixels enables a bright background image to bereconstructed. Therefore, distance measurement is possible even in abright illuminance environment, the distance information associated withthe image can be obtained, and a bright background image can also beobtained.

In this case, if the signal component is amplified by mixing of thepixels located around the pixel of interest in the same frame as thepixel of interest, a bright background image can be reconstructedwithout motion blur even when an imaged object is moving rapidly.

If the signal component is amplified by mixing of the pixels located atthe same position as the pixel of interest in the frames preceding andfollowing the frame including the pixel of interest, loss of staticresolution can be reduced, so that even for an imaged object having finepatterns and a complex contours, loss of resolution can be minimized anda bright, clear background image can be reconstructed.

The use of the optical filter 13 with transmittance in the visiblewavelength band set to the reciprocal of the number of summed pixels(the amplification factor) of the pixel summation unit 74 enables thereconstruction of a background image having substantially the samebrightness as before being attenuated by the optical filter 13. Togetherwith this, an imaging signal with a reduced ambient light component inthe visible wavelength band which interferes with extraction of thepattern light component can be obtained, enabling highly preciseextraction of the pattern light component, allowing distance measurementeven in a bright environment, and making it possible to obtain thedistance information in association with the image.

The fourth embodiment has been described as a variation of the thirdembodiment, but the features described in the fourth embodiment may alsobe added to either the first or second embodiment.

The first, second, third, and fourth embodiments use a laser as thelight source in the projection unit 22. But similar operation andsimilar effects can be obtained by use of some other type of lightsource instead, such as an LED, as long as the incident lightcharacteristics of the diffraction grating are satisfied.

In the first, second, third, and fourth embodiments, the projection unit22 having a configuration in which a pattern formed in the diffractiongrating 26 is projected by the laser light source 23 has been described.But similar operation and similar effects can also be obtained from aconfiguration that projects a pattern by scanning a laser beamtwo-dimensionally at a high speed (the entire field of view beingscanned within one frame period).

The first, second, third, and fourth embodiments use a diffractiongrating as an element for forming pattern light. But similar operationand similar effects can also be obtained by use of a spectral patternprojection device such as, for example, the transmissive computergenerated hologram described in paragraphs 0003 and 0004 in patentreference 3.

As described above, according to the present invention, an imagegeneration device is obtained with which information about the distanceto an object present in an imaged space can be obtained in associationwith the captured image. In addition, only one imaging element need beused. The image generation device according to the invention cansimultaneously acquire, for example, an image of an intruder and thedistance to the intruder, so that it can be used to detect intrusion inmonitoring applications. The image generation device according to theinvention is also applicable to driving assistance, such as parkingassistance, by the detection of obstacles in front of and behind avehicle.

REFERENCE CHARACTERS

10 image acquisition unit, 11 imaging unit, 12 lens, 13 optical filter,14 imaging element, 15 A/D conversion unit, 16 separation unit, 17 imagegeneration unit, 18 distance information generation unit, 19 displayprocessing unit, 20 pattern light generation unit, 21 drive unit, 22projection unit, 23 laser light source, 24 collimating lens, 25aperture, 26 diffraction grating, 30 control unit, 61 frame delay unit,62 difference calculation unit, 63 switch, 72 image signal processingunit, 74 pixel summation unit, 81 binarization unit, 82 spot areaextraction unit, 83 identification code reading unit, 84 storage unit,85 validation unit, 86 projection angle estimation unit, 87 incidenceangle calculation unit, 88 distance calculation unit, 89 distance rangecalculation unit.

1. An image generation device comprising: a projection unit for projecting pattern light of a prescribed wavelength into an imaged space; an imaging unit for imaging the imaged space; a separation unit for separating a projected pattern image component from an imaging signal obtained by imaging by the imaging unit by taking a difference between the imaging signal obtained when the pattern light is projected and the imaging signal obtained when the pattern light is not projected; and a distance information generation unit for generating distance information on a basis of a projected image component separated by the separation unit; wherein the distance information generation unit determines projection angles of light spots in the imaged projected pattern from an arrangement of the light spots in the projected image represented by the projected image component and a prestored relationship between the projection angles and positions of the light spots in the projected pattern, and determines a distance to a surface of an imaged object onto which the light spots are projected on a basis of the projection angles thus determined; the pattern light projected from the projection unit includes a plurality of cells, each in an on state or an off state, forming an identification code accompanying each light spot; the distance information generation unit determines the positions of the light spots accompanied by the identification codes in the projected pattern on a basis of the identification codes; the identification code accompanying each light spot has a first part comprising a plurality of cells, aligned in a first direction in the projected pattern, and disposed on one side of the light spot in a second direction in the projected pattern, and a second part comprising a plurality of cells, aligned in the second direction, and disposed on one side of the light spot in the first direction; the identification codes accompanying the light spots that are adjacent in the first direction have at most one location at which the cells constituting the second parts of the identification codes change from the on state to the off state or from the off state to the on state; and the first parts of the identification codes accompanying the light spots that are adjacent in the second direction are mutually identical; wherein the imaging unit has an optical filter having transmission characteristics in a visible wavelength band and a wavelength of the pattern light, with transmittance in the visible wavelength band being lower than at the wavelength of the pattern light; the separation unit separates the projected image component from the imaging signal obtained by imaging by the imaging unit by taking a difference between the imaging signal obtained when the pattern light is projected and the imaging signal obtained when the pattern light is not projected; and the transmission characteristics of the optical filter are determined such that the difference has a value equal to or greater than a prescribed number of gradation levels of the imaging signal.
 2. The image generation device of claim 1, wherein the distance information generation unit has: an identification code reading unit for reading the identification codes accompanying the light spots included in the projected image component obtained by the separation unit; a validation unit for determining whether or not a condition that there is at most one location where the second part of an identification code accompanying each light spot read by the identification code reading unit changes with respect to the second part of the identification code accompanying a light spot adjacent to said each light spot in the first direction, and that the first part of the identification code accompanying each light spot read by the identification code reading unit is identical to the first part of the identification code accompanying a light spot adjacent to said each light spot in the second direction, is satisfied or not, and thereby determining whether the identification code that is read is valid or not; and a projection angle estimation unit for estimating the projection angles of the light spots on a basis of the identification codes determined to be valid by the validation unit.
 3. An image generation device comprising: a projection unit for projecting pattern light of a prescribed wavelength into an imaged space; an imaging unit for imaging the imaged space; a separation unit for separating a projected image component from an imaging signal obtained by imaging by the imaging unit by taking a difference between the imaging signal obtained when the pattern light is projected and the imaging signal obtained when the pattern light is not projected; and a distance information generation unit for generating distance information on a basis of a projected image component separated by the separation unit; wherein the distance information generation unit determines projection angles of light spots in the imaged projected pattern from an arrangement of the light spots in the projected image represented by the projected image component and a prestored relationship between the projection angles and positions of the light spots in the projected pattern, and determines a distance to a surface of an imaged object onto which the light spots are projected on a basis of the projection angles thus determined; the pattern light projected from the projection unit includes a plurality of cells, each in an on state or an off state, forming an identification code accompanying each light spot; the distance information generation unit determines the positions of the light spots accompanied by the identification codes in the projected pattern on a basis of the identification codes; and the identification codes are determined in such a manner that there is only one location at which the cells constituting the identification codes change from the on state to the off state or from the off state to the on state between the light spots that are adjacent in a first direction in the projected pattern; wherein the imaging unit has an optical filter having transmission characteristics in a visible wavelength band and a wavelength of the pattern light, with transmittance in the visible wavelength band being lower than at the wavelength of the pattern light; the separation unit separates the projected image component from the imaging signal obtained by imaging by the imaging unit by taking a difference between the imaging signal obtained when the pattern light is projected and the imaging signal obtained when the pattern light is not projected; and the transmission characteristics of the optical filter are determined such that the difference has a value equal to or greater than a prescribed number of gradation levels of the imaging signal
 4. The image generation device of claim 3, wherein the identification code accompanying each light spot has a first part comprising a plurality of cells, aligned in a first direction in the projected pattern, and disposed on one side of the light spot in a second direction in the projected pattern, and a second part comprising a plurality of cells, aligned in the second direction, and disposed on one side of the light spot in the first direction.
 5. The image generation device of claim 4, wherein the distance information generation unit has an identification code reading unit for reading the identification codes accompanying the light spots included in the projected image component obtained by the separation unit; a validation unit for determining whether or not a condition that there is at most one location where the identification code accompanying each light spot read by the identification code reading unit changes with respect to the identification code accompanying a light spot adjacent in the first direction, and thereby determining whether the read identification code is valid or not; and a projection angle estimation unit for estimating the projection angles of the light spots on a basis of the identification codes determined to be valid by the validation unit.
 6. (canceled)
 7. The image generation device of claim 1, wherein the identification codes are determined such that an arrangement of cells in the on state and the off state constituting the identification codes accompanying all of the light spots included in the projected pattern is point symmetric with respect to the center of the projected pattern.
 8. (canceled)
 9. The image generation device of claim 7, wherein: the distance information generation unit calculates a range of distances to the imaged object on which the light spot is projected from a ratio between a size of the spot area in the projected image that has been imaged and a size of a dot positioned in the cell in the spot area; and when the distance to the imaged object calculated on the basis of the projection angle is not within the range of distances determined on a basis of the ratio between the size of the spot area and the size of the dot, the distance information generation unit treats as invalid the distance calculated on the basis of the projection angle. 10-13. (canceled)
 14. The image generation device of claim 1, wherein the separation unit separates, from the imaging signal obtained by imaging by the imaging unit, the imaging signal obtained when the pattern light is not projected, as a background component, said image generation device further comprising a background image generation unit for generating a background image from the background component separated by the separation unit.
 15. The image generation device of claim 14, wherein the background image generation unit has a pixel summation unit for amplifying the imaging signal by adding pixel values of surrounding pixels.
 16. The image generation device of claim 15, wherein the pixel summation unit adds, to each pixel, the pixel values of pixels in surrounding positions in the same frame.
 17. The image generation device of claim 15, wherein the pixel summation unit adds, to each pixel, pixel values of pixels in the same positions as the pixel in frames positioned preceding and following the frame including the pixel.
 18. (canceled)
 19. The image generation device of claim 15, wherein the transmittance of the optical filter in the visible wavelength band is equal to a reciprocal of the number of pixels summed by the pixel summation unit.
 20. The image generation device of claim 3, wherein the identification codes are determined such that an arrangement of cells in the on state and the off state constituting the identification codes accompanying all of the light spots included in the projected pattern is point symmetric with respect to the center of the projected pattern.
 21. The image generation device of claim 20, wherein: the distance information generation unit calculates a range of distances to the imaged object on which the light spot is projected from a ratio between a size of the spot area in the projected image that has been imaged and a size of a dot positioned in the cell in the spot area; and when the distance to the imaged object calculated on the basis of the projection angle is not within the range of distances determined on a basis of the ratio between the size of the spot area and the size of the dot, the distance information generation unit treats as invalid the distance calculated on the basis of the projection angle.
 22. The image generation device of claim 3, wherein the separation unit separates, from the imaging signal obtained by imaging by the imaging unit, the imaging signal obtained when the pattern light is not projected, as the background component, said image generation device further comprising a background image generation unit for generating a background image from a background component separated by the separation unit.
 23. The image generation device of claim 22, wherein the background image generation unit has a pixel summation unit for amplifying the imaging signal by adding pixel values of surrounding pixels.
 24. The image generation device of claim 23, wherein the pixel summation unit adds, to each pixel, the pixel values of pixels in surrounding positions in the same frame.
 25. The image generation device of claim 23, wherein the pixel summation unit adds, to each pixel, pixel values of pixels in the same positions as the pixel in frames positioned preceding and following the frame including the pixel.
 26. The image generation device of claim 23, wherein the transmittance of the optical filter in the visible wavelength band is equal to a reciprocal of the number of pixels summed by the pixel summation unit. 